Pandas Profiling Report On California house data

Dataset statistics

Number of variables	9
Number of observations	20640
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	1.4 MiB
Average record size in memory	72.0 B

Variable types

Numeric	9

Warnings

`Latitude` is highly correlated with `Longitude`	High correlation
`Longitude` is highly correlated with `Latitude`	High correlation
`AveRooms` is highly skewed (γ1 = 20.69786896)	Skewed
`AveBedrms` is highly skewed (γ1 = 31.31695625)	Skewed
`AveOccup` is highly skewed (γ1 = 97.63956096)	Skewed

Reproduction

Analysis started	2021-04-20 08:41:34.546695
Analysis finished	2021-04-20 08:42:05.594776
Duration	31.05 seconds
Software version	pandas-profiling v2.11.0
Download configuration	config.yaml

MedInc
Real number (ℝ_≥0)

Distinct	12928
Distinct (%)	62.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	3.870671003
Minimum	0.4999
Maximum	15.0001
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	0.4999
5-th percentile	1.60057
Q1	2.5634
median	3.5348
Q3	4.74325
95-th percentile	7.300305
Maximum	15.0001
Range	14.5002
Interquartile range (IQR)	2.17985

Descriptive statistics

Standard deviation	1.899821718
Coefficient of variation (CV)	0.4908249026
Kurtosis	4.952524102
Mean	3.870671003
Median Absolute Deviation (MAD)	1.0642
Skewness	1.646656702
Sum	79890.6495
Variance	3.60932256
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
15.0001	49	0.2%
3.125	49	0.2%
2.875	46	0.2%
2.625	44	0.2%
4.125	44	0.2%
3.875	41	0.2%
3.375	38	0.2%
3	38	0.2%
4	37	0.2%
3.625	37	0.2%
Other values (12918)	20217	98.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.4999	12	0.1%
0.536	10	< 0.1%
0.5495	1	< 0.1%
0.6433	1	< 0.1%
0.6775	1	< 0.1%
0.6825	1	< 0.1%
0.6831	1	< 0.1%
0.696	1	< 0.1%
0.6991	1	< 0.1%
0.7007	1	< 0.1%

Value	Count	Frequency (%)
15.0001	49	0.2%
15	2	< 0.1%
14.9009	1	< 0.1%
14.5833	1	< 0.1%
14.4219	1	< 0.1%
14.4113	1	< 0.1%
14.2959	1	< 0.1%
14.2867	1	< 0.1%
13.947	1	< 0.1%
13.8556	1	< 0.1%

HouseAge
Real number (ℝ_≥0)

Distinct	52
Distinct (%)	0.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	28.63948643
Minimum	1
Maximum	52
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	1
5-th percentile	8
Q1	18
median	29
Q3	37
95-th percentile	52
Maximum	52
Range	51
Interquartile range (IQR)	19

Descriptive statistics

Standard deviation	12.58555761
Coefficient of variation (CV)	0.4394477408
Kurtosis	-0.8006288536
Mean	28.63948643
Median Absolute Deviation (MAD)	10
Skewness	0.0603306376
Sum	591119
Variance	158.3962604
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
52	1273	6.2%
36	862	4.2%
35	824	4.0%
16	771	3.7%
17	698	3.4%
34	689	3.3%
26	619	3.0%
33	615	3.0%
18	570	2.8%
25	566	2.7%
Other values (42)	13153	63.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	4	< 0.1%
2	58	0.3%
3	62	0.3%
4	191	0.9%
5	244	1.2%
6	160	0.8%
7	175	0.8%
8	206	1.0%
9	205	1.0%
10	264	1.3%

Value	Count	Frequency (%)
52	1273	6.2%
51	48	0.2%
50	136	0.7%
49	134	0.6%
48	177	0.9%
47	198	1.0%
46	245	1.2%
45	294	1.4%
44	356	1.7%
43	353	1.7%

AveRooms
Real number (ℝ_≥0)

SKEWED

Distinct	19392
Distinct (%)	94.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	5.428999742
Minimum	0.8461538462
Maximum	141.9090909
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	0.8461538462
5-th percentile	3.432330006
Q1	4.440716236
median	5.229128788
Q3	6.052380952
95-th percentile	7.640246547
Maximum	141.9090909
Range	141.0629371
Interquartile range (IQR)	1.611664716

Descriptive statistics

Standard deviation	2.474173139
Coefficient of variation (CV)	0.4557327789
Kurtosis	879.353264
Mean	5.428999742
Median Absolute Deviation (MAD)	0.8029206051
Skewness	20.69786896
Sum	112054.5547
Variance	6.121532724
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
5	27	0.1%
4.5	22	0.1%
4	21	0.1%
6	20	0.1%
5.333333333	13	0.1%
5.5	11	0.1%
4.666666667	9	< 0.1%
3	8	< 0.1%
5.666666667	8	< 0.1%
7	7	< 0.1%
Other values (19382)	20494	99.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.8461538462	1	< 0.1%
0.8888888889	1	< 0.1%
1	1	< 0.1%
1.130434783	2	< 0.1%
1.260869565	1	< 0.1%
1.378486056	1	< 0.1%
1.411290323	1	< 0.1%
1.465753425	1	< 0.1%
1.550408719	1	< 0.1%
1.553030303	1	< 0.1%

Value	Count	Frequency (%)
141.9090909	1	< 0.1%
132.5333333	1	< 0.1%
62.42222222	1	< 0.1%
61.8125	1	< 0.1%
59.875	1	< 0.1%
56.26923077	1	< 0.1%
52.84821429	1	< 0.1%
52.69047619	1	< 0.1%
50.83783784	1	< 0.1%
47.51515152	1	< 0.1%

AveBedrms
Real number (ℝ_≥0)

SKEWED

Distinct	14233
Distinct (%)	69.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1.09667515
Minimum	0.3333333333
Maximum	34.06666667
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	0.3333333333
5-th percentile	0.9391087789
Q1	1.006079046
median	1.048780488
Q3	1.099526066
95-th percentile	1.273005718
Maximum	34.06666667
Range	33.73333333
Interquartile range (IQR)	0.09344702031

Descriptive statistics

Standard deviation	0.4739108568
Coefficient of variation (CV)	0.4321342167
Kurtosis	1636.711972
Mean	1.09667515
Median Absolute Deviation (MAD)	0.04608687412
Skewness	31.31695625
Sum	22635.37509
Variance	0.2245915002
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
1	288	1.4%
1.125	29	0.1%
1.058823529	26	0.1%
1.083333333	25	0.1%
1.1	25	0.1%
1.052631579	23	0.1%
1.090909091	21	0.1%
1.05	21	0.1%
1.055555556	20	0.1%
1.111111111	18	0.1%
Other values (14223)	20144	97.6%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.3333333333	1	< 0.1%
0.375	1	< 0.1%
0.4444444444	1	< 0.1%
0.5	3	< 0.1%
0.5263157895	1	< 0.1%
0.53125	1	< 0.1%
0.5454545455	1	< 0.1%
0.5555555556	1	< 0.1%
0.5625	2	< 0.1%
0.5714285714	2	< 0.1%

Value	Count	Frequency (%)
34.06666667	1	< 0.1%
25.63636364	1	< 0.1%
15.3125	1	< 0.1%
14.11111111	1	< 0.1%
11.41071429	1	< 0.1%
11.18181818	1	< 0.1%
11	1	< 0.1%
10.27027027	1	< 0.1%
10.15384615	1	< 0.1%
9.703703704	1	< 0.1%

Population
Real number (ℝ_≥0)

Distinct	3888
Distinct (%)	18.8%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1425.476744
Minimum	3
Maximum	35682
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	3
5-th percentile	348
Q1	787
median	1166
Q3	1725
95-th percentile	3288
Maximum	35682
Range	35679
Interquartile range (IQR)	938

Descriptive statistics

Standard deviation	1132.462122
Coefficient of variation (CV)	0.7944444737
Kurtosis	73.55311639
Mean	1425.476744
Median Absolute Deviation (MAD)	440
Skewness	4.935858227
Sum	29421840
Variance	1282470.457
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
891	25	0.1%
761	24	0.1%
1227	24	0.1%
850	24	0.1%
1052	24	0.1%
825	23	0.1%
999	22	0.1%
782	22	0.1%
1005	22	0.1%
781	21	0.1%
Other values (3878)	20409	98.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
3	1	< 0.1%
5	1	< 0.1%
6	1	< 0.1%
8	4	< 0.1%
9	2	< 0.1%
11	1	< 0.1%
13	4	< 0.1%
14	3	< 0.1%
15	2	< 0.1%
17	2	< 0.1%

Value	Count	Frequency (%)
35682	1	< 0.1%
28566	1	< 0.1%
16305	1	< 0.1%
16122	1	< 0.1%
15507	1	< 0.1%
15037	1	< 0.1%
13251	1	< 0.1%
12873	1	< 0.1%
12427	1	< 0.1%
12203	1	< 0.1%

AveOccup
Real number (ℝ_≥0)

SKEWED

Distinct	18841
Distinct (%)	91.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	3.070655159
Minimum	0.6923076923
Maximum	1243.333333
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	0.6923076923
5-th percentile	1.872544757
Q1	2.429741148
median	2.818115654
Q3	3.282260924
95-th percentile	4.333416667
Maximum	1243.333333
Range	1242.641026
Interquartile range (IQR)	0.8525197767

Descriptive statistics

Standard deviation	10.38604956
Coefficient of variation (CV)	3.382356215
Kurtosis	10651.01064
Mean	3.070655159
Median Absolute Deviation (MAD)	0.4195255877
Skewness	97.63956096
Sum	63378.32249
Variance	107.8700255
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
3	35	0.2%
2	18	0.1%
2.5	17	0.1%
2.666666667	16	0.1%
2.333333333	13	0.1%
2.6	12	0.1%
3.2	11	0.1%
2.555555556	10	< 0.1%
2.75	9	< 0.1%
3.25	9	< 0.1%
Other values (18831)	20490	99.3%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.6923076923	1	< 0.1%
0.75	1	< 0.1%
0.9705882353	1	< 0.1%
1.060606061	1	< 0.1%
1.066176471	1	< 0.1%
1.089267803	1	< 0.1%
1.089285714	1	< 0.1%
1.161290323	1	< 0.1%
1.169329073	1	< 0.1%
1.215873016	1	< 0.1%

Value	Count	Frequency (%)
1243.333333	1	< 0.1%
599.7142857	1	< 0.1%
502.4615385	1	< 0.1%
230.1724138	1	< 0.1%
83.17142857	1	< 0.1%
63.75	1	< 0.1%
51.4	1	< 0.1%
41.21428571	1	< 0.1%
33.95294118	1	< 0.1%
21.33333333	1	< 0.1%

Latitude
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct	862
Distinct (%)	4.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	35.63186143
Minimum	32.54
Maximum	41.95
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	32.54
5-th percentile	32.82
Q1	33.93
median	34.26
Q3	37.71
95-th percentile	38.96
Maximum	41.95
Range	9.41
Interquartile range (IQR)	3.78

Descriptive statistics

Standard deviation	2.135952397
Coefficient of variation (CV)	0.05994501302
Kurtosis	-1.117759781
Mean	35.63186143
Median Absolute Deviation (MAD)	1.23
Skewness	0.4659530037
Sum	735441.62
Variance	4.562292644
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
34.06	244	1.2%
34.05	236	1.1%
34.08	234	1.1%
34.07	231	1.1%
34.04	221	1.1%
34.09	212	1.0%
34.02	208	1.0%
34.1	203	1.0%
34.03	193	0.9%
33.93	181	0.9%
Other values (852)	18477	89.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
32.54	1	< 0.1%
32.55	3	< 0.1%
32.56	10	< 0.1%
32.57	18	0.1%
32.58	26	0.1%
32.59	11	0.1%
32.6	9	< 0.1%
32.61	14	0.1%
32.62	13	0.1%
32.63	18	0.1%

Value	Count	Frequency (%)
41.95	2	< 0.1%
41.92	1	< 0.1%
41.88	1	< 0.1%
41.86	3	< 0.1%
41.84	1	< 0.1%
41.82	1	< 0.1%
41.81	2	< 0.1%
41.8	3	< 0.1%
41.79	1	< 0.1%
41.78	3	< 0.1%

Longitude
Real number (ℝ)

HIGH CORRELATION

Distinct	844
Distinct (%)	4.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-119.5697045
Minimum	-124.35
Maximum	-114.31
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	-124.35
5-th percentile	-122.47
Q1	-121.8
median	-118.49
Q3	-118.01
95-th percentile	-117.08
Maximum	-114.31
Range	10.04
Interquartile range (IQR)	3.79

Descriptive statistics

Standard deviation	2.003531724
Coefficient of variation (CV)	-0.01675618195
Kurtosis	-1.330152366
Mean	-119.5697045
Median Absolute Deviation (MAD)	1.28
Skewness	-0.297801208
Sum	-2467918.7
Variance	4.014139367
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
-118.31	162	0.8%
-118.3	160	0.8%
-118.29	148	0.7%
-118.27	144	0.7%
-118.32	142	0.7%
-118.28	141	0.7%
-118.35	140	0.7%
-118.36	138	0.7%
-118.19	135	0.7%
-118.25	128	0.6%
Other values (834)	19202	93.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-124.35	1	< 0.1%
-124.3	2	< 0.1%
-124.27	1	< 0.1%
-124.26	1	< 0.1%
-124.25	1	< 0.1%
-124.23	3	< 0.1%
-124.22	1	< 0.1%
-124.21	3	< 0.1%
-124.19	4	< 0.1%
-124.18	6	< 0.1%

Value	Count	Frequency (%)
-114.31	1	< 0.1%
-114.47	1	< 0.1%
-114.49	1	< 0.1%
-114.55	1	< 0.1%
-114.56	1	< 0.1%
-114.57	3	< 0.1%
-114.58	2	< 0.1%
-114.59	2	< 0.1%
-114.6	3	< 0.1%
-114.61	3	< 0.1%

MedHouseVal
Real number (ℝ_≥0)

Distinct	3842
Distinct (%)	18.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	2.068558169
Minimum	0.14999
Maximum	5.00001
Zeros	0
Zeros (%)	0.0%
Memory size	161.4 KiB

Quantile statistics

Minimum	0.14999
5-th percentile	0.662
Q1	1.196
median	1.797
Q3	2.64725
95-th percentile	4.8981
Maximum	5.00001
Range	4.85002
Interquartile range (IQR)	1.45125

Descriptive statistics

Standard deviation	1.153956159
Coefficient of variation (CV)	0.55785531
Kurtosis	0.3278702429
Mean	2.068558169
Median Absolute Deviation (MAD)	0.684
Skewness	0.9777632739
Sum	42695.04061
Variance	1.331614816
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
5.00001	965	4.7%
1.375	122	0.6%
1.625	117	0.6%
1.125	103	0.5%
1.875	93	0.5%
2.25	92	0.4%
3.5	79	0.4%
0.875	78	0.4%
2.75	65	0.3%
1.5	64	0.3%
Other values (3832)	18862	91.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.14999	4	< 0.1%
0.175	1	< 0.1%
0.225	4	< 0.1%
0.25	1	< 0.1%
0.266	1	< 0.1%
0.269	1	< 0.1%
0.275	1	< 0.1%
0.283	1	< 0.1%
0.3	2	< 0.1%
0.325	4	< 0.1%

Value	Count	Frequency (%)
5.00001	965	4.7%
5	27	0.1%
4.991	1	< 0.1%
4.99	1	< 0.1%
4.988	1	< 0.1%
4.987	1	< 0.1%
4.986	1	< 0.1%
4.984	1	< 0.1%
4.976	1	< 0.1%
4.974	1	< 0.1%

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Count
Matrix

A simple visualization of nullity by column.

Nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

First rows

	MedInc	HouseAge	AveRooms	AveBedrms	Population	AveOccup	Latitude	Longitude	MedHouseVal
0	8.3252	41.0	6.984127	1.023810	322.0	2.555556	37.88	-122.23	4.526
1	8.3014	21.0	6.238137	0.971880	2401.0	2.109842	37.86	-122.22	3.585
2	7.2574	52.0	8.288136	1.073446	496.0	2.802260	37.85	-122.24	3.521
3	5.6431	52.0	5.817352	1.073059	558.0	2.547945	37.85	-122.25	3.413
4	3.8462	52.0	6.281853	1.081081	565.0	2.181467	37.85	-122.25	3.422
5	4.0368	52.0	4.761658	1.103627	413.0	2.139896	37.85	-122.25	2.697
6	3.6591	52.0	4.931907	0.951362	1094.0	2.128405	37.84	-122.25	2.992
7	3.1200	52.0	4.797527	1.061824	1157.0	1.788253	37.84	-122.25	2.414
8	2.0804	42.0	4.294118	1.117647	1206.0	2.026891	37.84	-122.26	2.267
9	3.6912	52.0	4.970588	0.990196	1551.0	2.172269	37.84	-122.25	2.611

Last rows

	MedInc	HouseAge	AveRooms	AveBedrms	Population	AveOccup	Latitude	Longitude	MedHouseVal
20630	3.5673	11.0	5.932584	1.134831	1257.0	2.824719	39.29	-121.32	1.120
20631	3.5179	15.0	6.145833	1.141204	1200.0	2.777778	39.33	-121.40	1.072
20632	3.1250	15.0	6.023377	1.080519	1047.0	2.719481	39.26	-121.45	1.156
20633	2.5495	27.0	5.445026	1.078534	1082.0	2.832461	39.19	-121.53	0.983
20634	3.7125	28.0	6.779070	1.148256	1041.0	3.026163	39.27	-121.56	1.168
20635	1.5603	25.0	5.045455	1.133333	845.0	2.560606	39.48	-121.09	0.781
20636	2.5568	18.0	6.114035	1.315789	356.0	3.122807	39.49	-121.21	0.771
20637	1.7000	17.0	5.205543	1.120092	1007.0	2.325635	39.43	-121.22	0.923
20638	1.8672	18.0	5.329513	1.171920	741.0	2.123209	39.43	-121.32	0.847
20639	2.3886	16.0	5.254717	1.162264	1387.0	2.616981	39.37	-121.24	0.894

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Missing values

Sample

First rows

Last rows